The effects of speech recognition and punctuation on information extraction performance

We report on experiments to measure the effect of speech recognition errors and automatic punctuation insertion errors on the performance of information extraction (entity and relation extraction). The outputs of several recognition systems with a range of word error rates (WER), along with punctuation insertion, were fed into a system that extracts entities and relations from the recognized text. Entity and relation value scores were measured as a function of WER and types of punctuation used. The results of the experiments showed that both entity and relation value scores degrade linearly with increasing WER, with a relative reduction in scores of about twice the WER. The information extraction modules require the inclusion of sentence boundaries, at a minimum; however, the experiments showed that the exact locations of these boundaries are not important for entity and relation extraction. In contrast, when comparing the effects of full punctuation to just automatic sentence boundary insertion, there was a loss in entity value scores of 13.5% and in relation value scores of 25%. Further, commas play a significantly greater role in entity and relation extraction than other types of punctuation.